Search for: All records

Digital experiments are routinely used to test the value of a treatment relative to a status quo control setting — for instance, a new search relevance algorithm for a website or a new results layout for a mobile app. As digital experiments have become increasingly pervasive in organizations and a wide variety of research areas, their growth has prompted a new set of challenges for experimentation platforms. One challenge is that experiments often focus on the average treatment effect (ATE) without explicitly considering differences across major sub-groups — heterogeneous treatment effect (HTE). This is especially problematic because ATEs have decreased in many organizations as the more obvious benefits have already been realized. However, questions abound regarding the pervasiveness of user HTEs and how best to detect them. We propose a framework for detecting and analyzing user HTEs in digital experiments. Our framework combines an array of user characteristics with double machine learning. Analysis of 27 real-world experiments spanning 1.76 billion sessions and simulated data demonstrates the effectiveness of our detection method relative to existing techniques. We also find that transaction, demographic, engagement, satisfaction, and lifecycle characteristics exhibit statistically significant HTEs in 10% to 20% of our real-world experiments, underscoring the importance of considering user heterogeneity when analyzing experiment results, otherwise personalized features and experiences cannot happen, thus reducing effectiveness. In terms of the number of experiments and user sessions, we are not aware of any study that has examined user HTEs at this scale. Our findings have important implications for information retrieval, user modeling, platforms, and digital experience contexts, in which online experiments are often used to evaluate the effectiveness of design artifacts. 

This research examines the roles of health literacy, health numeracy, and trust in doctor on: (1) patient anxiety when consulting a doctor; (2) frequency of physician consultations; and (3) patient subjective well‐being (SWB). Our sample consisted of 4,040 adults representative of the United States in terms of age, income, and education, but equally split among White/Caucasian (50%) and Black/African American (50%) respondents. We found that functional and communicative health literacy and trust in doctor have linear and curvilinear relationships with anxiety when consulting a doctor. Health numeracy had no effect. Anxiety when consulting a doctor was associated with a lower number of physician consultations and lower SWB. We observed direct linear effects of health literacy, health numeracy, and trust in doctor on frequency of physician consultations and SWB, as well as some curvilinear effects. We found a negative relationship between health numeracy and SWB. We discuss implications of these findings for health and public policy.